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The findings from this review do not reflect the full body of research evidence 
on text message financial aid reminders. 


What is this study about? 

The study measured the impact of sending text mes- 
sage reminders regarding annual Free Application for 
Federal Student Aid (FAFSA) renewal to first-year col- 
lege students who were already receiving financial aid. 
Specifically, the study authors measured the effect of 
these messages on persistence into the second year 
of college. The text messages (a) provided information 
about how the students could obtain help with finan- 
cial aid, (b) reminded students about important dead- 
lines and requirements for refilling the FAFSA, and (c) 
offered assistance on financial aid related processes. 
Students in the comparison group did not receive text 
message reminders, but did have financial aid process 
assistance available to them on request (in-person 
assistance rates were similar in the intervention and 
comparison groups). The study authors state that the 
intervention costs about $5 per student. 

Most of the students participating in this study 
were attending a postsecondary institution in Mas- 
sachusetts. All of the participants were first-time 
college students who had received services during 
high school from the host organization (uAspire, 
a national nonprofit that provides financial aid 
counseling). About 62% of the participants were 
female, 91 % were members of a racial/ethnic minor- 
ity group, and 83% were eligible for Pell Grants. 
Seventy-two percent of students attended a 4-year 
intuition during their freshman year. 


What did the study find? 

The study authors reported that, overall, there was no 
statistically significant effect on the rates of student 
persistence from their freshman to their sophomore 
years (persistence rates were about 81 % in both 
groups). Fiowever, the intervention’s effect differed for 
students attending 2-year vs. 4-year institutions. While 
the persistence rates were similar among students 
attending a 4-year institution (84% in the intervention 
group and 87% in the comparison group), students 
attending a 2-year institution who participated in the 
intervention persisted at a higher rate (75%) than did 
students in the comparison condition (64%). 2 


Features of Text Message Reminders 


Students received text messages approximately 
every 2 weeks. The text messages focused on 
financial aid renewal by: 

• offering assistance with the financial aid process, 

• reminding students of important deadlines, and 

• reminding students of the need to maintain 
satisfactory academic progress. 


WWC Rating 


The research described in this 
report meets WWC group design 
standards without reservations 

This study is a well-executed randomized controlled 
trial. 
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Appendix A: Study details 

Castleman, B. L., & Page, L. C. (2014). Freshman year financial aid nudges: An experiment to increase 
FAFSA renewal and college persistence (EdPolicyWorks Working Paper Series No. 29). 
Charlottesville, VA: EdPolicyWorks. 


Setting 

The students in this sample were attending postsecondary institutions, most of which were 


located in Massachusetts. 

Study sample 

The sample included 808 students. Most of the students participating in this study were 
attending a postsecondary institution in Massachusetts. All of the participants were first-time 
college students who were receiving financial aid. About 62% of the participants were female, 
91 % were members of a racial/ethnic minority group, and 83% were eligible for Pell Grants. All 
of the students in the sample had received services during high school from the host organiza- 
tion (uAspire, a national nonprofit organization that provides financial aid counseling). To be 
included in the study, students must have had a cell phone number on file with uAspire and 
must have enrolled in college for the fall 2012 semester. 

Intervention 

group 

The intervention group in the study was comprised of students who were randomly assigned 
to receive the text message reminders. 

Comparison 

group 

The comparison group in the study was comprised of students who were randomly assigned 
not to receive the text message reminders. These students were eligible for support from 
uAspire on request. 

Outcomes and 
measurement 

The study reported findings for one eligible outcome— persistence from the first year to the 
second year of college. The outcome measure was based on data from the National Student 
Clearinghouse. 

Support for 
implementation 

The authors did not provide information on training received by uAspire counselors. 

Reason for 
review 

This study was reviewed by the WWC in response to receiving significant media attention. 
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Appendix B: Outcome measure for the credit accumulation domain 


Credit accumulation 


Persistence This outcome is measured via a binary indicator of whether or not the student re-enrolled in college for the fall 

semsester of the second year, obtained from the National Student Clearinghouse. 

Study Notes: This study also assessed the impact of text message reminders on several indicators related to intervention take up and on whether students re-enrolled at the same 
institution. None of these outcomes were eligible for review under the Postsecondary Education review protocol, and therefore, are not included in this single study review. 
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Appendix C: Study findings for the credit accumulation domain 


Mean 

(standard deviation) WWC calculations 


Domain and 
outcome measure 

Study 

sample 

Sample 

size 

Intervention 

group 

Comparison 

group 

Mean 

difference 

Effect 

size 

Improvement 

index 

p-value 

Credit accumulation 

Persistence 

Postsecondary 

students 

808 

students 

81% 

81% 

1% 

+0.02 

+1 

> .05 

Domain average for credit accumulation 





+0.02 

+1 

Not 


statistically 

significant 


Table Notes: For mean difference, effect size, and improvement index values reported in the table, a positive number favors the intervention group and a negative number favors 
the comparison group. The effect size is a standardized measure of the effect of an intervention on individual outcomes, representing the average change expected for all individu- 
als who are given the intervention (measured in standard deviations of the outcome measure). The improvement index is an alternate presentation of the effect size, reflecting 
the change in an average individual's percentile rank that can be expected if the individual is given the intervention. The WWC-computed average effect size is a simple average 
rounded to two decimal places; the average improvement index is calculated from the average effect size. The statistical significance of the study’s domain average was deter- 
mined by the WWC; for example, the study is characterized as having a statistically significant positive effect because univariate statistical tests are reported for each outcome 
measure, the effect for at least one measure within the domain is positive and statistically significant, and no effects are negative and statistically significant, accounting for 
multiple comparisons. Some statistics may not sum as expected due to rounding. 

Study Notes: No corrections for clustering or multiple comparisons and no difference-in-differences adjustment were needed. The p-value presented here was reported in the 
original study. Unadjusted standard error estimates were obtained from the study authors. These statistics form the basis of the effect size calculation. Some statistics may not 
sum as expected due to rounding. 
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Appendix D: Supplemental findings for the credit accumulation domain 


Mean 

(standard deviation) WWC calculations 


Domain and 
outcome measure 

Study 

sample 

Sample 

size 

Intervention 

group 

Comparison 

group 

Mean 

difference 

Effect 

size 

Improvement 

index 

p-value 

Credit accumulation 

Persistence 

4-year 

institution 

583 students 

84% 

87% 

-3% 

-0.11 

-4 

> .05 

Persistence 

2-year 

225 students 

75% 

64% 

11% 

0.28 

+11 

< .05 


institution 


Table Notes: The supplemental findings presented in this table are additional findings that do not factor into the determination of the evidence rating. For mean difference, effect 
size, and improvement index values reported in the table, a positive number favors the intervention group and a negative number favors the comparison group. The effect size 
is a standardized measure of the effect of an intervention on individual outcomes, representing the average change expected for all individuals who are given the intervention 
(measured in standard deviations of the outcome measure). The improvement index is an alternate presentation of the effect size, reflecting the change in an average individual's 
percentile rank that can be expected if the individual is given the intervention. 

Study Notes: No corrections for clustering or multiple comparisons and no difference-in-differences adjustments were needed. The p-values presented here were reported in the 
original study. Using data obtained from the study authors, tests for subgroup interaction revealed a statistically significant difference in effects across subgroups. 
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Endnotes 

1 Single study reviews examine evidence published in a study (supplemented, if necessary, by information obtained directly from the 
authors) to assess whether the study design meets WWC design standards. The review reports the WWC’s assessment of whether 
the study meets WWC design standards and summarizes the study findings following WWC conventions for reporting evidence on 
effectiveness. This study was reviewed using the Postsecondary Education topic area review protocol, version 3.0. The WWC rating 
applies only to the study outcomes that were eligible for review under this topic area. The reported analyses in this SSR are only for 
those eligible outcomes that either met WWC design standards without reservations or met WWC design standards with reservations, 
and do not necessarily apply to all results presented in the study. 

2 There were outcomes included in the study that are not described in this WWC report— several indicators related to intervention take 
up, and on whether students re-enrolled at the same institution. See the study notes in Appendix B for more information. 

Recommended Citation 

U.S. Department of Education, Institute of Education Sciences, What Works Clearinghouse. (2014, December). 

WWC review of the report: Freshman year financial aid nudges: An experiment to increase FAFSA renewal and 
college persistence. Retrieved from http://whatworks.ed.gov 
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Glossary of Terms 

Attrition 


Clustering adjustment 
Confounding factor 

Design 
Domain 
Effect size 

Eligibility 

Equivalence 

Improvement index 


Multiple comparison 
adjustment 

Quasi-experimental 
design (QED) 

Randomized controlled 
trial (RCT) 

Single-case design 
(SCD) 

Standard deviation 


Statistical significance 
Substantively important 


Attrition occurs when an outcome variable is not available for all participants initially assigned 
to the intervention and comparison groups. The WWC considers the total attrition rate and 
the difference in attrition rates across groups within a study. 

If intervention assignment is made at a cluster level and the analysis is conducted at the student 
level, the WWC will adjust the statistical significance to account for this mismatch, if necessary. 

A confounding factor is a component of a study that is completely aligned with one of the 
study conditions, making it impossible to separate how much of the observed effect was 
due to the intervention and how much was due to the factor. 

The design of a study is the method by which intervention and comparison groups were assigned. 
A domain is a group of closely related outcomes. 

The effect size is a measure of the magnitude of an effect. The WWC uses a standardized 
measure to facilitate comparisons across studies and outcomes. 

A study is eligible for review if it falls within the scope of the review protocol and uses either 
an experimental or matched comparison group design. 

A demonstration that the analysis sample groups are similar on observed characteristics 
defined in the review area protocol. 

Along a percentile distribution of individuals, the improvement index represents the gain 
or loss of the average individual due to the intervention. As the average individual starts at 
the 50th percentile, the measure ranges from -50 to +50. 

When a study includes multiple outcomes or comparison groups, the WWC will adjust 
the statistical significance to account for the multiple comparisons, if necessary. 

A quasi-experimental design (QED) is a research design in which study participants are 
assigned to intervention and comparison groups through a process that is not random. 

A randomized controlled trial (RCT) is an experiment in which eligible study participants are 
randomly assigned to intervention and comparison groups. 

A research approach in which an outcome variable is measured repeatedly within and 
across different conditions that are defined by the presence or absence of an intervention. 

The standard deviation of a measure shows how much variation exists across observations 
in the sample. A low standard deviation indicates that the observations in the sample tend 
to be very close to the mean; a high standard deviation indicates that the observations in 
the sample are spread out over a large range of values. 

Statistical significance is the probability that the difference between groups is a result of 
chance rather than a real difference between the groups. The WWC labels a finding statistically 
significant if the likelihood that the difference is due to chance is less than 5% (p < .05). 

A substantively important finding is one that has an effect size of 0.25 or greater, regardless 
of statistical significance. 


Please see the WWC Procedures and Standards Handbook (version 3.0) for additional details. 
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